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ABSTRACT 

Two expert system development projects were stud- 
ied to evaluate a proposed Expert Systems Develop- 
ment Methodology (ESDM). The ESDM was devel- 
oped for use at Goddard Space Flight Center (GSFC) 
to provide guidance to managers and technical per- 
sonnel and serve as a standard in the development of 
expert systems. It was agreed that the proposed 
ESDM must be evaluated before it could be adopted; 
therefore a study was planned for its evaluation. This 
detailed study is now underway. Before the study 
began, however, two ongoing projects were selected 
for a retrospective evaluation. They were the Rang- 
ing Equipment Diagnostic Expert System (REDEX) 
and the Backup Control Mode Analysis and Utility 
System (BCAUS). Both projects were approximately 
1 year into development. Interviews of project per- 
sonnel were conducted, and the resulting data was 
used to prepare the retrospective evaluation. Deci- 
sion models of the two projects were constructed and 
used to evaluate the completeness and accuracy of 
key provisions of ESDM. A major conclusion reached 
from these case studies is that suitability and risk 
analysis should be required for all AI projects, large 
and small. Further, the objectives of each stage of 
development during a project should be selected to 
reduce the next largest area of risk or uncertainty on 
the project. 

INTRODUCTION 

The Expert Systems Development Methodology 
(ESDM) is intended to be applied to the development 
of expert systems at the National Aeronautics and 
Space Administration/Goddard Space Flight Center 
(NASA/GSFC). The methodology is based on a sur- 
vey of existing methodologies, experience in develop- 
ing a number of expert systems at GSFC, and an 
analysis of the expert system life cycle. Dr. Barry W. 
Boehm introduced a risk-driven methodology for 
conventional systems development in his spiral 
model for software development (Boehm, 1988). 
ESDM, while independently generated, is also a 
risk-driven methodology that can be represented by a 


spiral model with the focus on knowledge acquisition 
as opposed to product development. Figure 1 shows 
the spiral model of ESDM. 

Risks are inherent in all system development proj- 
ects, but they are greater in ES development because 
of the uncertainties associated with modeling human 
expert decision processes. At the outset of the devel- 
opment of an expert system, it is not known whether 
an expert’s decision processes are cognitive processes 
that can be modeled by ES techniques. Some human 
decisions are made on the basis of intuition or skills, 
which usually cannot be modeled using ES tech- 
niques. Intuitive processes and skills can often be 
modeled using other techniques, such as neural net- 
works, but ESDM does not address these. Even after 
it has been determined that an expert’s decision proc- 
esses can be modeled, there remain developmental 
risks because of uncertainties about the robustness 
and performance that can be obtained from the 
expert system. 

ESDM was developed as a tool for both project man- 
agers and developers of expert systems in the NASA 
environment. It focuses on the knowledge acquisition 
task, rather than on product development. Key fea- 
tures and recommendations of ESDM include: 

• Decomposition of an ES development proj- 
ect into stages. In each stage, work is directed 
toward the acquisition of the key knowledge 
needed to reduce the most immediate or 
highest level risk of the project. 

• Explicit identification of the objectives of 
each stage of work prior to its initiation and 
testing to verify that the obj ectives have been 
met. 

• Well-defined criteria for stopping ES devel- 
opment. Once the functional requirements 
of the proposed system have been identified, 
ESDM recommends dropping the ES 
approach and continuing the project along 
the lines of conventional system develop- 
ment. ESDM also recommends stopping the 
ES project if the expert’s decision processes 
are not suitable for ES modeling or if an 
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algorithm is discovered that performs the 
decision process satisfactorily. 

• ESDM provides guidance for the kinds of 
special documentation needed for ES proj- 
ects and the management information that 
should be collected for administrative 
reporting. 

• ESDM recommends using of quantitative 
methods for assessing risk where possible 
and provides a tool, the Test for Application 
of Risk-Oriented Technology (TAROT), to 
assist in this evaluation. 

The ESDM project has produced a user’s guide 
(CSCa, 1988), a policy document (CSCb, 1988), and a 
reference manual (CSCc, 1988) along with training 
materials. ESDM has been proposed for use on all 
GSFC expert system projects. Because a proposed 
methodology must be evaluated before its adoption, 
however, a framework for the evaluation was also 
developed (CSC, 1989). The framework recom- 
mended selecting an expert system development 


project and using it as a pilot to evaluate the features 
of the ESDM before its adoption as a standard. The 
project would be followed from beginning to end and 
would collect data on ESDM effectiveness. 

Before undertaking a full-scale pilot study, two on- 
going projects were selected for a retrospective eval- 
uation of ESDM. The Ranging Equipment Diagnos- 
tic Expert System (REDEX) and the Backup Control 
Mode Analysis and Utility Systems (BCAUS) were 
the two projects selected. Data on the two projects 
was collected by interviewing project personnel. De- 
cision models of the two projects were also con- 
structed and used to evaluate the completeness and 
accuracy of ESDM in accordance with the general 
provisions of the framework for evaluation. 

This paper presents a summary of the findings made 
on these two case studies. The case studies include a 
description of the two projects, the key decisions 
made on the projects, and conclusions reached about 
the methodology. 
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THE REDEX SYSTEM 

REDEX is an advanced prototype expert system that 
diagnoses hardware failures in the ranging equip- 
ment (RE) at NASA’s Ground Network tracking sta- 
tions (Luczak, 1989). REDEX is intended for use by 
RE technicians in identifying faulty circuit cards or 
modules that must be replaced. The system has a 
highly graphical user interface that uses color block 
and layout diagrams to illustrate fault locations. 
Figure 2 shows the environment for REDEX. 

The REDEX project was initiated by the Telecommu- 
nication Systems Branch (Code 531) at GSFC as a 
task assignment. There were two persons assigned to 
the project initially, but the level of effort has 
averaged less than two full-time persons. 

No formal risk or suitability analysis of the project was 
performed. The use of an expert system as a diagnos- 
tic aid for the RE was considered feasible because the 
RE had been designed with a large number of built-in 
test points. It was expected that these test points 
would greatly facilitate the automation of fault diag- 
nosis, and the task of REDEX was to speed up the 
identification process. 

Development staff personnel were generally familiar 
with the provisions of ESDM. On their own initiative, 
they selected ESDM features that they believed 
would assist them in the development of REDEX and 
used them in the project. The selected features were: 

• The use of a staged development 

• The decomposition of stages into steps 


• The use of risk analysis to guide the selection 
of objectives for stages 

The stages of work on REDEX followed ESDM rec- 
ommendations closely for addressing successively 
more complex objectives. Five stages of work were 
defined, each addressing more complex issues. The 
following summarizes these five stages: 

1. Feasibility of implementing one diagnostic 
rule and accomplishing diagnosis with this 
rule 

2. Feasibility of extending the feasibility proto- 
type to include all relevant rules on the 
selected hardware host (IBM PC-AT) 

3. Feasibility of implementing one graphics 
screen on the selected host 

4. Feasibility of extending the graphics system 
to include all required graphics 

5. The capability of the system to be fielded 
(field prototype), including handling all 
necessary communications with the equip- 
ment 

REDEX is implemented in Prolog on an IBM PC 
AT-compatible workstation. A semantic network 
knowledge representation technique was used to 
model the design structure of the RE. A catalog of 
generic troubleshooting rules was compiled to repre- 
sent heuristics that are applied in diagnosis. Specific 
troubleshooting rules unique to the RE were also 
added. Over 50 generic and 250 specific rules were 
developed. A hypertext-like scheme is used to allow 
the user to navigate through the diagrams and tables. 
Over 50 graphic and tabular displays have been 
implemented. 



Figure 2. NASA Ground Network Tracking Station 
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The client identified the use of IBM PC -AT hardware 
as desirable at the outset of the project. The choice of 
computer language or shell development system was 
left to project personnel, but an early identification of 
the language was requested for budgetary estimation. 
At project initiation, it was not clear what knowledge 
representation scheme might be most appropriate for 
the knowledge yet to be acquired, and project person- 
nel began addressing this problem immediately. 
After investigation, they determined that a rules 
representation would be appropriate. Generic and 
specific rules (in a pseudocode form) for the system 
were manually compiled. 

From the somewhat limited choices available for the 
implementation of knowledge-based systems on an 
IBM PC-AT, project personnel concluded, after fur- 
ther investigation of six languages and shell develop- 
ment systems, that Prolog appeared to be suitable for 
prototyping the rules. A feasibility prototype was then 
developed to test whether Prolog was suitable for the 
system prototypes. (Prolog’s suitability for the opera- 
tional system was not determined at this time; how- 
ever, its suitability in this regard has since been 
proved.) At the completion of the second prototype 
stage, called the research stage in ESDM, the func- 
tional requirements for REDEX were validated. 
These requirements were then documented and 
issued in a functional requirements document. 

The fifth (and current) prototype stage of REDEX 
addresses those risks associated with using the system 
in the field. In this prototype, the uncertainties are 
concerned with the communications between the RE 
and REDEX. This system is currently being 
evaluated in a communications emulation testbed 
and will be connected to the RE after evaluation. 

THE BCAUS SYSTEM 

BCAUS is an expert system designed to assist flight 
operations personnel in diagnosing the cause of a 
Gamma Ray Observatory (GRO) spacecraft autono- 
mous mode transition (Bush, 1989). The GRO space- 
craft was designed with onboard capability to safe 
itself autonomously, transitioning from a primary op- 
erating mode to a backup control (safing) mode in the 
event of certain error conditions in the attitude con- 
trol and determination (ACAD) subsystem. 

Flight operations personnel need to understand what 
error condition trigger the onboard computer (OBC) 
to order the mode transition and why that error con- 
dition occurred so that they may take the proper cor- 
rective action. The OBC was not designed, however, 
to provide the triggering information or the diag- 
nostic information to the operator. 


Input information to BCAUS will be provided by 
telemetry data from GRO and by user input. Output 
from BCAUS will be provided only to the diagnosti- 
cian. There is no output back to the spacecraft. 
Figure 3 shows a diagram of the information flows in 
BCAUS. 

GSFC also initiated the BCAUS project by issuing a 
task assignment. Two persons were assigned to the 
project. No formal risk or suitability analysis of 
BCAUS was performed. Task personnel had knowl- 
edge of the risk areas in expert system development 
and used this information to guide the development 
process. The primary area of risk for BCAUS was in 
the knowledge acquisition process. Four sources of 
expertise were identified and were initially consid- 
ered adequate for the development task. These four 
sources were (1) documentation, (2) GSFC space- 
craft design experts, (3) GSFC flight operations 
experts, and (4) TRW personnel associated with the 
design of the relevant GRO subsystems. However, 
project personnel found that the knowledge acquisi- 
tion task for this system was more difficult than ini- 
tially thought and that the initial evaluation of risk 
had to be modified. The project goals have therefore 
shifted from providing an operational system to a 
system in which the knowledge base is easily modified 
and updated on the basis of actual experience. In 
brief, the goal has shifted from providing an initially 
operational system to an adaptive system with an 
initial base of knowledge that can be upgraded as 
expertise is acquired. 

The difficulty in the knowledge acquisition task for 
the BCAUS project is that the expertise needed to 
diagnose GRO mode transitions has not yet been 
acquired by humans. There was no training course 
available for GRO fault diagnosis as there was for 
REDEX. The existence of a training course means 
that the diagnostic knowledge has been compiled, 
thus implying a lower risk of system development. 
However, even the designers of the GRO subsystems 
had not yet acquired or compiled all the information 
necessary for mode transition analysis, and this fact 
was not known at the outset of the project. The rela- 
tive inaccessibility of the TRW design engineers 
because of their location on the west coast and their 
limited availability for consultation made it difficult 
for project personnel to elicit any available informa- 
tion. When the full difficulty of knowledge acquisi- 
tion became known, a reevaluation and reorientation 
of project goals and objectives became necessaiy. The 
complexity of the knowledge acquisition task on the 
BCAUS project perhaps doubled the time required 
to reach a feasibility prototype. This situation con- 
strained the design of the operational system in ways 
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Figure 3. The BCAUS System 


that were not and probably could not have been, 
determined at the outset. 

The first prototype system, a feasibility prototype, was 
implemented on a PC-386 class machine using the 
KES hypothesize-and-test (HT) inference engine 


developed by Software Architecture and Engi- 
neering, Inc. Basic structural knowledge of the sys- 
tem elements was loaded on the machine in three 
weeks by two persons. No rules were needed because 
of the built-in diagnostic feature of KES HT. When 
KES HT was selected initially, there were some 
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known limitations on its capabilities. Following its use 
on the project, it became clear that the limitations 
were too restrictive, especially in the area of explana- 
tory power. The second prototyping system selected 
was ARX-IM, an expert system development shell 
produced by Inference Corporation, which can run on 
both the 386 machine and a Silicon Graphics Iris 
4D/20. ITte Iris has considerably more power for 
graphics than the 386 machine. 

The BCAUS system will also have a neural network 
front end to provide trending data on input telemetry 
signals. Project personnel determined that it might be 
possible to implement trending analysis using 
AKT-IM rules, but at the expense of making the sys- 
tem much larger and more complex than desirable. 
The software product. Neural Works Professional II, 
from Neural Ware, Inc., was selected to implement 
the trending analysis. 

The BCAUS system’s graphics interface shows rele- 
vant subsystems in the form of functional block dia- 
grams, similar to those implemented in REDEX, 
with highlighted potential problem areas. A hierar- 
chical traverse is planned for navigation of the dia- 
grams and causal graphs. 

As was the case with REDEX, there was a strong 
tendency to follow the methodology used for conven- 
tional software systems, and the hardware and soft- 
ware selections were set very early in the project. The 
deadlines for hardware and software selection were 
met only through very concentrated effort on the part 
of the project’s development staff. 

PROJECT KEY DECISIONS 

In the evaluation process, ESDM was modeled as a 
sequence of key decisions plus subsidiary decisions. 
Key decisions are identified on the basis of their pos- 
sible impacts on project cost and schedule. The key 
decisions of ESDM are: 

• Start. The decision that the project is suit- 
able for implementation by an expert system 
is based on a formal suitability analysis in 
ESDM. 

• Knowledge-oriented approach. Is current 
knowledge about the problem sufficient to 
permit preparation of specifications for the 
system now? If not, then a knowledge- 
oriented approach is indicated, that is, the 
decision is made to acquire the missing infor- 
mation first. 


• Staffing. ESDM recommends a knowledge 
engineer, AI programmers, system program- 
mers, and domain experts for expert system 
projects. 

• Staging. ESDM recommends dividing a proj- 
ect into successive stages based on the 
degree of uncertainty about how to accom- 
plish the function or service. 

• Steps within stages. ESDM recommends fol- 
lowing five steps within each stage. These 
steps focus on identifying the knowledge to 
be acquired within the stage, on acquiring 
this knowledge, and on verifying the correct- 
ness of the acquired knowledge. 

• Explicit risk evaluation. ESDM recommends 
that all risk evaluation be explicit, that is, 
that each area of risk on the project be docu- 
mented and assessed. ESDM also provides a 
formal tool, the TAROT metric, to assist in 
estimating of risks. 

• Stop-rule. This decision is based on acquiring 
sufficient information to prepare meaning- 
ful and realizable specifications. At this 
point, ESDM recommends continuing the 
project as a normal software development 
project following the conventional software 
development life cycle. 

There are also technical and managerial decisions of 
lesser importance that have some bearing on project 
schedule and costs: 

• Reporting (frequency, type, content) 

• Hardware and software tools selection and 
timing 

• Use of automated knowledge tools 

• Need for graphics and interfaces 

The method used to evaluate ESDM was, first, rating 
how closely the circumstances and decisions of the 
two projects matched the provisions of ESDM and, 
second, assessing the worth of the provisions based on 
the experience gained on the projects. 

The start decision on both REDEX and BCAUS, that 
is, the decision to use an expert system or knowledge- 
based technology to develop a system was made by 
GSFC personnel, not by the development project 
personnel. There is no information on whether any 
formal analysis of suitability was made by GSFC per- 
sonnel. In retrospect, it is clear that both projects 
were, in fact, suitable. By now, the usefulness of 
expert systems for fault diagnosis has been well estab- 
lished; this fact can be considered generally well 
known in the computer field. 

The ESDM provision that recommends a suitability 
analysis for each new project should be amended to 


344 



take into account current practices, informal stan- 
dards, and common knowledge among practitioners 
in the computer field. The study concluded that a 
more formal analysis of suitability should still be per- 
formed for any system that does not fall into one of 
the familiar categories of expert system applications. 
The decision regarding a formal suitability analysis on 
any new project is a judgment call. Nevertheless, 
ESDM must continued to provide the guidelines and 
procedures for cases requiring a suitability analysis. 

It was apparent even from a casual analysis of both 
REDEX and BCAUS that it was impossible to pre- 
pare specifications at the outset for either project and 
that a knowledge acquisition process would be 
required. What is important, however, is that knowl- 
edge acquisition procedures are required and that 
identifying the missing pieces of information is neces- 
sary in order to develop the systems. The identifica- 
tion of this information was carried out on both 
projects. 

Both the REDEX and BECAUS projects were 
staffed with experienced AI professionals. ESDM 
guidelines call for both knowledge engineers and AI 
programmers. Because of the small size of the proj- 
ects, however, it was necessary for project personnel 
to function both as knowledge engineers and as AI 
programmers. Also, project personnel assumed some 
of the functions of domain experts ESDM should be 
modified to take the special requirements of small 
projects into account, but the need for experienced 
and competent staff personnel becomes even more 
acute for these smaller projects. Managers should 
remain aware of the staffing requirement differences 
in small and large projects. 

Both REDEX and BCAUS were decomposed into 
successive stages of work. ESDM recommends defin- 
ing stages in terms of risk and addressing areas of 
highest risk first. While there was no conscious deci- 
sion to follow ESDM provisions on selecting stages, 
REDEX personnel nevertheless followed the feasi- 
bility, research, and field stages quite faithfully. 
REDEX also decomposed the planned system into 
three subsystems (functional, user interface, and 
communications interface) and followed the' risk- 
reduction sequencing in each subsystem. Staging was 
also followed on BCAUS. The first year of work on 
BCAUS was considered to be the feasibility stage. 

ESDM defines the research stage as that stage of work 
that establishes that one or a small set of rules can be 
implemented. The issue to be addressed then is 
whether enough of the required rules can be imple- 
mented to make the system practical. A better name 
for this stage should reflect the intent of the stage, 


that is, determining how far the feasibility prototype 
can be extended. The name, extensibility stage , has 
been suggested as a replacement. 

Both projects followed some natural sequence of 
work within the stages that was similar to the steps 
described in ESDM. In fact, the steps within the 
stages recommended in ESDM are a paraphrase of 
the scientific method, which is the model for knowl- 
edge acquisition or discovery processes. 

There was no formal analysis of risks made on either 
REDEX or BCAUS; however, both development 
teams reported being acutely aware of the risks asso- 
ciated with different areas of their projects at all 
times and stated that their work was governed by this 
awareness. This awareness of risk characterizes the 
experience of the development teams. Less experi- 
enced personnel might not have the opportunity to 
put together workable and useful systems. 

On small projects, there is less need for formal analy- 
sis of risk. The lack of a formal analysis on small 
projects should not be a concern to managers, as long 
as the staff is aware of risks and is guided by their 
consideration. On large projects, the use of a formal 
risk analysis is still recommended. ESDM provisions 
are being modified to take the size of the project into 
account. 

There are no plans to transfer REDEX or BCAUS to 
a conventional development cycle after preparation 
of system requirements. On small NASA projects, 
the personnel who began the project will typically 
carry on the development even after requirements 
have been specified and risks reduced to an accept- 
able levels. Transfer to a conventional life cycle with a 
new development team, which was recommended in 
ESDM for large projects, will probably be the excep- 
tion, rather than the rule, for most small projects. 

The documentation prepared on the two projects 
tended to follow the requirements for conventional 
software development. Although it is impossible to 
draw conclusions about ESDM provisions for docu- 
mentation, project personnel felt that the knowledge 
acquisition process was not adequately documented 
by the normal system development documents, thus 
lending support to the ESDM provisions. Some 
adjustment for the number of documents recom- 
mended by ESDM should be made on the basis of 
project size. 

Similar to the typical requirements associated with 
conventional software engineering, the hardware and 
software tools to be used on a project must be speci- 
fied at the outset or as early as possible. Based on the 
two retrospective studies, the conclusion is that 
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ESDM provisions are the least risky, that is, delaying 
the selection of hardware and software tools until 
after identification of the knowledge structure. 

Because of the experiences gained by project person- 
nel on REDEX and BCAUS, they suggested the pos- 
sibility of using automated tools for logging and man- 
aging lists of rules or other knowledge structures. 
Since the final evaluation is incomplete at this time, 
there has been no change in ESDM regarding recom- 
mendations on the use of automated tools. 

In addition to the two projects described in this study, 
there was a review of a number of other expert sys- 
tems prepared for NASA. Nearly all of them made 
use of color graphics interfaces for presentation of 
information to the users. This fact has some implica- 
tions for the selection of tools for the development of 
expert systems and the selection of personnel to work 
on expert systems. A possible modification to ESDM 
will point this out and provide guidelines for tool and 
personnel selection. 

REDEX makes use of input data from equipment 
monitoring points and BCAUS has telemetry inputs 
from the GRO spacecraft. Many NASA expert sys- 
tems have sources of input information other than 
the human. Also, many make use of other techniques 
than logic programming, such as: 

• Neural networks 

• Procedural code 

• Operating system calls 

The staffing requirements provisions of ESDM that 
address only knowledge engineering and AI program- 
ming should be modified to take into account the 
possible needs for systems programming, neural net 
programming, and familiarity with telemetry and 
communications. 


CONCLUSIONS 

The two projects surveyed match the model of the 
expert system development life cycle so closely that 
the experience gained on these projects provides 
valuable information for ESDM evaluation. The two 
projects are quite different in detail and dynamics, 
and they differ from the expected large-size project 
envisioned by ESDM. The experience of these proj- 
ects is useful primarily in providing ESDM with 
extensions to cover the cases of small-size projects. 


General conclusions reached from the retrospective 
study of the two projects include: 

• Confirmation of the need for a methodology. 
The standard systems development method- 
ology matches the life cycle of expert systems 
poorly. The need for a methodology better 
suited to the special requirements of expert 
systems is supported by project experience. 

• Support for the use of a risk-based approach. 
Both project teams reported that they were 
aware of risks in development and organized 
their projects to address these risks. ESDM 
formalizes this practice in ES development. 

• The decomposition of projects into succes- 
sive stages. Both projects broke the work 
down into successive stages to make the 
overall task more manageable. 

• Requirements as an overall goal. Both proj- 
ects produced requirements documents at 
the conclusion of an extensive knowledge 
acquisition phase in accordance with the rec- 
ommendations of ESDM. 

Based on the evaluations provided by the two proj- 
ects, REDEX and BCAUS, it was possible to reach 
some specific conclusions about the details of ESDM 
and the framework to be used for its evaluation on the 
two pilot projects. 

• ESDM currently requires a formal suitability 
analysis for all projects. Findings suggest 
that this requirement should be relaxed for 
small projects. 

• ESDM should be modified to describe the 
differences between small and large proj- 
ects. In particular, some of the formal docu- 
ments required for projects are unnecessary 
for small projects and may impose an unnec- 
essary burden. 

• The name of the research stage of the 
ESDM life cycle should be changed to the 
extensibility stage. 

• The use of the TAROT metric (or other for- 
mal tool) for evaluation of risk and the suit- 
ability of candidate projects for ESDM 
should be optional for small projects. 

• Personnel qualifications for expert systems 
development should include experience and 
familiarity with graphics user interfaces as 
well as with the functional tools required for 
expert systems. 
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